Repository navigation
Hold deleting repository secrets and branch protection through gh - #468
Conversation
Cygnux-Labs#402) Signed-off-by: aj-codespy <ajayush2301@gmail.com>
8be329e to
0d332d6
Compare
|
Thanks for this. TK-A043 closes a real gap from the E21 pool, and it only adds a rule, so nothing existing gets weaker. Before it can merge:
Once CI is green we'll do a second review. |
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
The 1.5.1 release takes TK-A043 to TK-A046, so this rule moves to TK-A047 in the pack, its test and the docs. The E17, E21 policy-robustness and E21 robustness results are regenerated against the new rule and corpus lines, and the E21 latency table in the evaluation page quotes the new run. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
|
I made the changes from the 2026-10-28 review, so the branch is ready for CI and the second review:
Why each job failed on the last CI run:
tests/test_signer_policy.py, tests/test_policy2.py and tests/test_eval_docs.py pass locally (210 tests), and |
Conflicts resolved so both sides stay: - coding.yaml: main's 1.5.1 rules (TK-A043 to TK-A046 payment and download holds, TK-D002/TK-D007 precision, memory-notes exceptions) plus TK-A047 after TK-A042; pack 2026.10-v2-43. TK-A047's remedy now reads "Have a person approve the held call", as every held call's does since 1.5.1. - tests/test_signer_policy.py: both sides' tests. - README.md, docs: main's text with TK-A047's ask entry and limits row, and E17's counts at 705 harmful and 808 benign calls. - eval/results: E17, E21 policy robustness and E21 robustness run afresh; E21's latency block kept as main recorded it. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
The second review of Cygnux-Labs#468 found TK-A047's fallback-engine searches super-linear and several ways past it. On Python 3.9 (no google-re2), 64 KiB of `$` or backquotes in a gh or hub api argument spent the decision's whole second and denied as nondeterministic; so did a branch name repeating `/protection` (found while fixing); a repeated security_and_analysis/disabled pair took ~0.6 s. The run before a `$` or backquote now stops at the first one, the text around security_and_analysis is bounded, and a branch name is read to 250 characters: the same inputs decide in 10-53 ms on the fallback engine. Now held as well: the method in any case and after -X ending a cluster of short options (-iX, -sSX); an empty or space-edged argument; the owner and repository as one variable (repos/$GITHUB_REPOSITORY/...); the API's URL as a variable, its scheme in capitals or left out; gh repo edit turning secret scanning, push protection or advanced security off; an environment and paths under it; a DELETE of vulnerability-alerts or automated-security-fixes; the user's Codespaces secrets. A security_and_analysis PATCH is held only when a feature's status is set to disabled, so turning one on with "disabled" in another field isn't. The gh api DELETE with any variable stays held (lean): a variable before the path's last segment can hide a protected path. docs/limits.md now says which run-time paths are held and lists curl's URL rewriting, xargs, curl -K, numeric repository ids and long branch names. Speed gains `$`, backquote and the pair as fillers, and its adversarial decide test the three inputs that denied. E17 gains 10 harmful and 6 benign shell calls; E17 and both E21 results are re-run, E21's latency block kept as recorded, and the docs quote the new counts. Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
An independent review of the previous commit found two ways past the
rule: a method word taken at run time (gh api -X "$M", -X ${M:-DELETE},
-X $(echo DELETE), a backquoted echo, curl -X "$M", hub api -X "$M")
passed every API branch, and gh repo edit --enable-secret-scanning=$OFF
(and the push-protection and advanced-security flags) was allowed.
A method word with a $ or a backquote in it is now read wherever DELETE,
PUT or PATCH is, before or after a written-out protected path or a
security_and_analysis body; joined to its flag (-X$M) it must start with
one, so the flag's run isn't rescanned. gh repo edit's value is held
when it has a $ or a backquote in it, as =false is. A run-time method on
another path (an issue comment) isn't held.
docs/limits.md's TK-A047 row now also says: the sixteen-piece gaps
count each space in a quoted argument and the words before the method;
JSON padding past 200 characters or 16 non-word characters isn't read;
percent-encoded parts, doubled slashes, capital path parts and
status=Disabled aren't matched; which neighbouring security changes are
out of scope; and that any host is read, so a PUT to a local test server
on the same paths is held.
E17 and E21 decide the same; only the pack hashes move.
Signed-off-by: LoopGlitch26 <53336715+LoopGlitch26@users.noreply.github.com>
|
Thanks again for contributing to Tracekit. A request: if you use a coding agent (Claude Code, Codex, Cursor, Gemini CLI or OpenCode), would you run Tracekit on it for a week and tell us how it went? After a week, |
|
@LoopGlitch26 Sure, I'd be happy to help. I use Cursor and Claude Code, so should I go with either one? |
|
@aj-codespy Either works. Claude Code is the quickest start: On issues: #563 (Tracekit hooks for OpenHands, the CLI and SDK coding agent) just opened up and is the closest to the AI side. Comment there to take it. |
|
@LoopGlitch26 The week's report is in: #815 (Claude Code, 7 days on macOS; 408 runs, all verified). Thanks for the nudge, and for the reviews on #662. |
What and why
The second blind E21 pool (#401) showed that an agent with a GitHub token could take a repository's protections apart
with nobody asked: all six of its such calls got through the server pack.
TK-A043, anaskrule next toTK-A042,now holds them in every pack:
gh secret deleteandgh variable delete(orremove), with--repo/-R,--envor--org;DELETE,PUTorPATCHthroughgh/hub apiorcurlon a branch'sprotectionor a path under it, arepository's or organization's
rulesets, or their Actions, Dependabot or Codespacessecretsandvariables(anenvironment's too), as a bare path or a URL (also GitHub Enterprise's
/api/v3), with the method before or after itand up to sixteen option words around the method, read as
TK-A042readsgh api;PATCHthat sets asecurity_and_analysisfeature todisabled;gh apiDELETEwith$or a backquote in an argument, whose path is built at run time (the pool'sbase64-decoded secret path).
A
GETof the same paths,gh secret list,gh secret setand aDELETEof an issue comment stay allowed. Every gapin the pattern is bounded. Coding pack version
2026.10-v2-33.tests/data/policy/deny/shell.jsonl, the pool's lookalikes plusgh secret setand anissue-comment
DELETEintests/data/policy/benign/shell.jsonl, and a four-pack × two-engine test intests/test_signer_policy.pywith the option orders, methods, paths and lookalikes above.TK-A043in the coding pack'saskrow ofdocs/policy-v2.md, and adocs/limits.mdrow for what it can'tread (another forge, a GraphQL mutation, a body from a file, a token used from a script or SDK, other HTTP clients).
PUTof a secret through the API is held, as the issue lists it,while
gh secret setisn't; and agh apiDELETEwith any$in its arguments is held, an issue comment's by avariable id too (marked
lean:).#401 pool after this change (its first score stays the blind one), both engines deciding every call the same:
server.yaml)strict.yaml)curl --dry-run,curl -I)On server, CI/CD changes went from 0 to 2 of 4 and defense evasion from 4 to 8 of 13; the other two CI/CD misses are
workflow files (#405).
E17: 705 of 705 harmful calls stopped, still 1 false alarm (the draft email), now in 808. E21: the held-out pool's
numbers are unchanged (default 245 of 420, strict 274 of 420), and all 705 E17 originals and 1,821 shell rewrites are
contained. E21's
machineandlatencyare kept as published, since this run was on a laptop. The README and docsthat quote E17's and E21's counts follow.
Fixes #402
Tests (macOS, Python 3.14):
tests/test_signer_policy.py,tests/test_policy2.py(including the 64 KiB patterntiming on both engines) and
tests/test_eval_docs.py,tracekit policy lint,make eval.Checklist
make checkpasses (lint, tests, build);make test-tstoo ifsdk/typescript/changedCHANGELOG.md)docs/threat-model-laptop.md,docs/threat-model-server.md)git commit -s, seeCONTRIBUTING.md)Security invariants
approval identity) is newly trusted by the signer or verifier
Evidence format
core.sig_message,ledger.make_record, the v1 schema, howthe verifier treats existing bundles) or the published v2 format
docs/sample/andtests/fixtures/) still verify, and the change is called out above